Vison-based object detection using a polar grid

ABSTRACT

A computing device of a first vehicle may receive a first image and a second image of a second vehicle having flashing light signals. The computing device may determine, in the first image and the second image, an image region that bounds the second vehicle such that the image region substantially encompasses the second vehicle. The computing device may determine a polar grid that partitions the image region in the first image and the second image into polar bins, and identify portions of image data exhibiting a change in color and a change in brightness between the first image and the second image. The computing device may determine a type of the flashing light signals and a type of the second vehicle; and accordingly provide instructions to control the first vehicle.

CROSS REFERENCE TO RELATED APPLICATION

The present application is a continuation of U.S. patent applicationSer. No. 15/671,316, filed on Aug. 8, 2017, and entitled “Vision-BasedObject Detection Using a Polar Grid,” which is a continuation of U.S.patent application Ser. No. 14/244,988, filed on Apr. 4, 2014, andentitled “Vision-Based Object Detection Using a Polar Grid,” whichissued as U.S. Pat. No. 9,766,628, the entire contents of all of whichare herein incorporated by reference as if fully set forth in thisdescription.

BACKGROUND

Autonomous vehicles use various computing systems to aid in transportingpassengers from one location to another. Some autonomous vehicles mayrequire some initial input or continuous input from an operator, such asa pilot, driver, or passenger. Other systems, for example autopilotsystems, may be used only when the system has been engaged, whichpermits the operator to switch from a manual mode (where the operatorexercises a high degree of control over the movement of the vehicle) toan autonomous mode (where the vehicle essentially drives itself) tomodes that lie somewhere in between.

SUMMARY

The present disclosure describes embodiments that relate to vision-basedobject detection using a polar grid. In one aspect, the presentdisclosure describes a method. The method includes receiving, at acomputing device of a first vehicle, a first image of a second vehiclecaptured by an image-capture device coupled to the first vehicle and asecond image captured by the image-capture device subsequent tocapturing the first image. The second vehicle has one or more flashinglight signals. The method also includes determining, in the first imageand the second image, an image region that bounds the second vehiclesuch that the image region substantially encompasses the second vehicle.The method further includes determining a polar grid that partitions theimage region in the first image and the second image into a plurality ofpolar bins, where each polar bin includes a sector of the image region.The method also includes identifying, based on a comparison of imagecontent of polar bins in the first image to image content ofcorresponding polar bins in the second image, one or more portions ofimage data exhibiting a change in color and a change in brightnessbetween the first image and the second image. The method furtherincludes determining a type of the one or more flashing light signals ofthe second vehicle and a type of the second vehicle based on (i) anumber of portions of image data exhibiting the change in color and thechange in brightness, (ii) the color of the one or more portions, (iii)and the brightness of the one or more portions. The method also includesproviding, by the computing device, instructions to control the firstvehicle based on the type of the second vehicle and the type of the oneor more flashing light signals.

In another aspect, the present disclosure describes a non-transitorycomputer readable medium having stored thereon executable instructionsthat, upon execution by a computing device of a first vehicle, cause thecomputing device to perform functions. The functions include receiving afirst image of a second vehicle captured by an image-capture devicecoupled to the first vehicle and a second image captured by theimage-capture device subsequent to capturing the first image, where thesecond vehicle has one or more flashing light signals. The functionsalso include determining, in the first image and the second image, animage region that bounds the second vehicle such that the image regionsubstantially encompasses the second vehicle. The functions furtherinclude determining a polar grid that partitions the image region in thefirst image and the second image into a plurality of polar bins, whereeach polar bin of the plurality of polar bins is defined by two linesextending from a center portion of the image region to about a boundaryof the image region. The functions also include identifying, based on acomparison of image content of polar bins in the first image to imagecontent of corresponding polar bins in the second image, one or moreportions of image data exhibiting a change in color and a change inbrightness between the first image and the second image. The functionsfurther include determining a type of the one or more flashing lightsignals of the second vehicle and a type of the second vehicle based on(i) a number of portions of image data exhibiting the change in colorand the change in brightness, (ii) the color of the one or moreportions, (iii) and the brightness of the one or more portions. Thefunctions also include providing instructions to control the firstvehicle based on the type of the second vehicle and the type of the oneor more flashing light signals.

In still another aspect, the present disclosure describes a system. Thesystem includes an image-capture device coupled to a first vehicle. Thesystem also includes at least one processor in communication with theimage-capture device. The system further includes a memory having storedthereon executable instructions that, upon execution by the at least oneprocessor, cause the system to perform functions. The functions comprisereceiving a first image of a second vehicle captured by theimage-capture device and a second image captured by the image-capturedevice subsequent to capturing the first image, where the second vehiclehas one or more flashing light signals. The functions also comprisedetermining, in the first image and the second image, an image regionthat bounds the second vehicle such that the image region substantiallyencompasses the second vehicle. The functions further comprisedetermining a polar grid that partitions the image region in the firstimage and the second image into a plurality of polar bins, where eachpolar bin of the plurality of polar bins is defined by two linesextending from a center portion of the image region to about a boundaryof the image region. The functions also comprise identifying, based on acomparison of image content of polar bins in the first image to imagecontent of corresponding polar bins in the second image, one or moreportions of image data exhibiting a change in color and a change inbrightness between the first image and the second image. The functionsfurther comprise determining a type of the one or more flashing lightsignals of the second vehicle and a type of the second vehicle based on(i) a number of portions of image data exhibiting the change in colorand the change in brightness, (ii) the color of the one or moreportions, (iii) and the brightness of the one or more portions. Thefunctions also comprise providing instructions to control the firstvehicle based on the type of the second vehicle and the type of the oneor more flashing light signals.

The foregoing summary is illustrative only and is not intended to be inany way limiting. In addition to the illustrative aspects, embodiments,and features described above, further aspects, embodiments, and featureswill become apparent by reference to the figures and the followingdetailed description.

BRIEF DESCRIPTION OF THE FIGURES

FIG. 1 is a simplified block diagram of an example automobile, inaccordance with an example embodiment.

FIG. 2 illustrates an example automobile, in accordance with an exampleembodiment.

FIG. 3 is a flow chart of a method for vision-based object detectionusing a polar grid, in accordance with an example embodiment.

FIG. 4 illustrates an image depicting an emergency vehicle, inaccordance with an example embodiment.

FIG. 5 illustrates an image region bounding the emergency vehicle, inaccordance with an example embodiment.

FIG. 6 illustrates a polar grid configured to partition the image regioninto a plurality of polar bins, in accordance with an exampleembodiment.

FIG. 7 illustrates example types of polar bins, in accordance with anexample embodiment.

FIG. 8 illustrates assigning a pixel within the image region to a polarbin, in accordance with an example embodiment.

FIG. 9 illustrates determining a number of flashing lights and a type ofthe emergency vehicle, in accordance with an example embodiment.

FIG. 10 illustrates other examples of determining types of vehicles, inaccordance with an example embodiment.

FIG. 11 is a schematic illustrating a conceptual partial view of acomputer program, in accordance with an example embodiment.

DETAILED DESCRIPTION

The following detailed description describes various features andfunctions of the disclosed systems and methods with reference to theaccompanying figures. In the figures, similar symbols identify similarcomponents, unless context dictates otherwise. The illustrative systemand method embodiments described herein are not meant to be limiting. Itmay be readily understood that certain aspects of the disclosed systemsand methods can be arranged and combined in a wide variety of differentconfigurations, all of which are contemplated herein.

I. Introduction

An autonomous vehicle operating on a road relies on detection ofobstacles or objects on the road for navigation. A computing device ofthe vehicle may be configured to detect and trace objects between imagesreceived from an image-capture device coupled to the vehicle. Forexample, the computing device may be configured to detect and trace apotential emergency vehicle (e.g., a police car, a fire truck, andambulance, etc.) across multiple images so as to make appropriatenavigation decisions. Detecting and tracing an emergency vehicle may,for example, rely on identifying flashing lights that characterizeemergency vehicles.

In an example, the computing device may determine an image region thatbounds a given object in a vicinity of the vehicle. For instance, thecomputing device may detect an object (e.g., any type of vehicle) andmay determine a bounding region, such as a bounding box, thatsubstantially encompasses the object in the image. The computing devicemay trace the object in a sequence of images. However, tracing theobject across the sequence of images may be difficult because, as thevehicle and the object move relative to each other, the position of theobject changes from one image to a subsequently captured image. Forinstance, the bounding box may not be determined accurately in eachimage (e.g., the object may be slightly to the side of the estimatedobject location in a given image). Thus, accuracy of tracing the objectin subsequent images may decrease because of position errors that mightaccumulate as the computing device detects the object in subsequentimages.

To enhance object detection and tracing, the computing device maytransform the rectangular bounding box into a polar grid (i.e., anangular region). The polar grid is configured to partition the boundingbox into a plurality of polar bins. Each polar bin may include a sectorof the bounding box. For instance, a polar bin may be defined by twolines extending from a center portion of the bounding box region toabout a boundary of the bounding box. The computing device may assigneach pixel in the image region within the bounding box to one of thepolar bins.

Further, the computing device may identify bright lights within eachpolar grid bin in a first image. For example, the computing device maydetermine a number of image portions that exhibit characteristics of abright light having an intensity greater than a threshold intensity. Thethreshold intensity may, for example, be indicative of a minimumintensity associated with flashing lights an emergency vehicle, avehicular turn signal, brake lights, etc. The computing device may alsoreceive a second image captured subsequent to capturing the first image.The computing device may partition a bounding box encompassing theobject in the second image into a polar grid corresponding to the polargrid in the first image. Thus, each polar bin in the polar grid of thefirst image has a corresponding polar bin in the polar grid of thesecond image. Within each corresponding polar bin in the second image,the computing device may identify image portions corresponding to theimage portions identified in the first image. The computing device maydetermine a change in color and a change in brightness for the imageportions from the polar bin in the first image to the correspondingpolar bin in the second image.

This process may be repeated to trace the image portions across multipleimages (e.g., a sequence of images). Based on comparing the polar binsacross multiple images, the computing device may determine whether agiven image portion within a given polar bin represents flashing lights(characterized by a repetitive or cyclical change in color or brightnessacross the images). Based on determining a number and types of flashinglight signals for the object, the computing device may determine a typeof the object (e.g., whether the object is a police vehicle having atleast two flashing light signals exhibiting a blue color, a fire truckhaving at least four flashing light signals, etc.) The computing devicemay provide instructions to control the vehicle based on the type of thevehicle (e.g., cause the vehicle to stop and allow a fire truck topass).

II. Example Systems

An example vehicle control system may be implemented in or may take theform of an automobile. Alternatively, a vehicle control system may beimplemented in or take the form of other vehicles, such as cars, trucks,motorcycles, buses, boats, airplanes, helicopters, lawn mowers,recreational vehicles, amusement park vehicles, farm equipment,construction equipment, trams, golf carts, trains, and trolleys. Othervehicles are possible as well.

Further, an example system may take the form of a non-transitorycomputer-readable medium, which has program instructions stored thereonthat are executable by at least one processor to provide thefunctionality described herein. An example system may also take the formof an automobile or a subsystem of an automobile that includes such anon-transitory computer-readable medium having such program instructionsstored thereon.

Referring now to the Figures, FIG. 1 is a simplified block diagram of anexample automobile 100, in accordance with an example embodiment.Components coupled to or included in the automobile 100 may include apropulsion system 102, a sensor system 104, a control system 106,peripherals 108, a power supply 110, a computing device 111, and a userinterface 112. The computing device 111 may include a processor 113, anda memory 114. The computing device 111 may be a controller, or part ofthe controller, of the automobile 100. The memory 114 may includeinstructions 115 executable by the processor 113, and may also store mapdata 116. Components of the automobile 100 may be configured to work inan interconnected fashion with each other and/or with other componentscoupled to respective systems. For example, the power supply 110 mayprovide power to all the components of the automobile 100. The computingdevice 111 may be configured to receive information from and control thepropulsion system 102, the sensor system 104, the control system 106,and the peripherals 108. The computing device 111 may be configured togenerate a display of images on and receive inputs from the userinterface 112.

In other examples, the automobile 100 may include more, fewer, ordifferent systems, and each system may include more, fewer, or differentcomponents. Additionally, the systems and components shown may becombined or divided in any number of ways.

The propulsion system 102 may be configured to provide powered motionfor the automobile 100. As shown, the propulsion system 102 includes anengine/motor 118, an energy source 120, a transmission 122, andwheels/tires 124.

The engine/motor 118 may be or include any combination of an internalcombustion engine, an electric motor, a steam engine, and a Stirlingengine. Other motors and engines are possible as well. In some examples,the propulsion system 102 could include multiple types of engines and/ormotors. For instance, a gas-electric hybrid car could include a gasolineengine and an electric motor. Other examples are possible.

The energy source 120 may be a source of energy that powers theengine/motor 118 in full or in part. That is, the engine/motor 118 maybe configured to convert the energy source 120 into mechanical energy.Examples of energy sources 120 include gasoline, diesel, otherpetroleum-based fuels, propane, other compressed gas-based fuels,ethanol, solar panels, batteries, and other sources of electrical power.The energy source(s) 120 could additionally or alternatively include anycombination of fuel tanks, batteries, capacitors, and/or flywheels. Insome examples, the energy source 120 may provide energy for othersystems of the automobile 100 as well.

The transmission 122 may be configured to transmit mechanical power fromthe engine/motor 118 to the wheels/tires 124. To this end, thetransmission 122 may include a gearbox, clutch, differential, driveshafts, and/or other elements. In examples where the transmission 122includes drive shafts, the drive shafts could include one or more axlesthat are configured to be coupled to the wheels/tires 124.

The wheels/tires 124 of automobile 100 could be configured in variousformats, including a unicycle, bicycle/motorcycle, tricycle, orcar/truck four-wheel format. Other wheel/tire formats are possible aswell, such as those including six or more wheels. The wheels/tires 124of automobile 100 may be configured to rotate differentially withrespect to other wheels/tires 124. In some examples, the wheels/tires124 may include at least one wheel that is fixedly attached to thetransmission 122 and at least one tire coupled to a rim of the wheelthat could make contact with the driving surface. The wheels/tires 124may include any combination of metal and rubber, or combination of othermaterials.

The propulsion system 102 may additionally or alternatively includecomponents other than those shown.

The sensor system 104 may include a number of sensors configured tosense information about an environment in which the automobile 100 islocated. As shown, the sensors of the sensor system include a GlobalPositioning System (GPS) module 126, an inertial measurement unit (IMU)128, a radio detection and ranging (RADAR) unit 130, a laser rangefinderand/or light detection and ranging (LIDAR) unit 132, a camera 134, andactuators 136 configured to modify a position and/or orientation of thesensors. The sensor system 104 may include additional sensors as well,including, for example, sensors that monitor internal systems of theautomobile 100 (e.g., an O₂ monitor, a fuel gauge, an engine oiltemperature, etc.). Other sensors are possible as well.

The GPS module 126 may be any sensor configured to estimate a geographiclocation of the automobile 100. To this end, the GPS module 126 mayinclude a transceiver configured to estimate a position of theautomobile 100 with respect to the Earth, based on satellite-basedpositioning data. In an example, the computing device 111 may beconfigured to use the GPS module 126 in combination with the map data116 to estimate a location of a lane boundary on road on which theautomobile 100 may be travelling on. The GPS module 126 may take otherforms as well.

The IMU 128 may be any combination of sensors configured to senseposition and orientation changes of the automobile 100 based on inertialacceleration. In some examples, the combination of sensors may include,for example, accelerometers and gyroscopes. Other combinations ofsensors are possible as well.

The RADAR unit 130 may be considered as an object detection system thatmay be configured to use radio waves to determine characteristics of theobject such as range, altitude, direction, or speed of the object. TheRADAR unit 130 may be configured to transmit pulses of radio waves ormicrowaves that may bounce off any object in a path of the waves. Theobject may return a part of energy of the waves to a receiver (e.g.,dish or antenna), which may be part of the RADAR unit 130 as well. TheRADAR unit 130 also may be configured to perform digital signalprocessing of received signals (bouncing off the object) and may beconfigured to identify the object.

Other systems similar to RADAR have been used in other parts of theelectromagnetic spectrum. One example is LIDAR (light detection andranging), which may be configured to use visible light from lasersrather than radio waves.

The LIDAR unit 132 may include a sensor configured to sense or detectobjects in an environment in which the automobile 100 is located usinglight. Generally, LIDAR is an optical remote sensing technology that canmeasure distance to, or other properties of, a target by illuminatingthe target with light. As an example, the LIDAR unit 132 may include alaser source and/or laser scanner configured to emit laser pulses and adetector configured to receive reflections of the laser pulses. Forexample, the LIDAR unit 132 may include a laser range finder reflectedby a rotating mirror, and the laser is scanned around a scene beingdigitized, in one or two dimensions, gathering distance measurements atspecified angle intervals. In examples, the LIDAR unit 132 may includecomponents such as light (e.g., laser) source, scanner and optics,photo-detector and receiver electronics, and position and navigationsystem.

In an example, The LIDAR unit 132 may be configured to use ultraviolet(UV), visible, or infrared light to image objects and can be used with awide range of targets, including non-metallic objects. In one example, anarrow laser beam can be used to map physical features of an object withhigh resolution.

In examples, wavelengths in a range from about 10 micrometers (infrared)to about 250 nm (UV) could be used. Typically light is reflected viabackscattering. Different types of scattering are used for differentLIDAR applications, such as Rayleigh scattering, Mie scattering andRaman scattering, as well as fluorescence. Based on different kinds ofbackscattering, LIDAR can be accordingly called Rayleigh LIDAR, MieLIDAR, Raman LIDAR and Na/Fe/K Fluorescence LIDAR, as examples. Suitablecombinations of wavelengths can allow for remote mapping of objects bylooking for wavelength-dependent changes in intensity of reflectedsignals, for example.

Three-dimensional (3D) imaging can be achieved using both scanning andnon-scanning LIDAR systems. “3D gated viewing laser radar” is an exampleof a non-scanning laser ranging system that applies a pulsed laser and afast gated camera. Imaging LIDAR can also be performed using an array ofhigh speed detectors and a modulation sensitive detectors arraytypically built on single chips using CMOS (complementarymetal-oxide-semiconductor) and hybrid CMOS/CCD (charge-coupled device)fabrication techniques. In these devices, each pixel may be processedlocally by demodulation or gating at high speed such that the array canbe processed to represent an image from a camera. Using this technique,many thousands of pixels may be acquired simultaneously to create a 3Dpoint cloud representing an object or scene being detected by the LIDARunit 132.

A point cloud may include a set of vertices in a 3D coordinate system.These vertices may be defined by X, Y, and Z coordinates, for example,and may represent an external surface of an object. The LIDAR unit 132may be configured to create the point cloud by measuring a large numberof points on the surface of the object, and may output the point cloudas a data file. As the result of a 3D scanning process of the object bythe LIDAR unit 132, the point cloud can be used to identify andvisualize the object.

In one example, the point cloud can be directly rendered to visualizethe object. In another example, the point cloud may be converted topolygon or triangle mesh models through a process that may be referredto as surface reconstruction. Example techniques for converting a pointcloud to a 3D surface may include Delaunay triangulation, alpha shapes,and ball pivoting. These techniques include building a network oftriangles over existing vertices of the point cloud. Other exampletechniques may include converting the point cloud into a volumetricdistance field and reconstructing an implicit surface so defined througha marching cubes algorithm.

The camera 134 may be any camera (e.g., a still camera, a video camera,etc.) configured to capture images of the environment in which theautomobile 100 is located. To this end, the camera may be configured todetect visible light, or may be configured to detect light from otherportions of the spectrum, such as infrared or ultraviolet light. Othertypes of cameras are possible as well. The camera 134 may be atwo-dimensional detector, or may have a three-dimensional spatial range.In some examples, the camera 134 may be, for example, a range detectorconfigured to generate a two-dimensional image indicating a distancefrom the camera 134 to a number of points in the environment. To thisend, the camera 134 may use one or more range detecting techniques. Forexample, the camera 134 may be configured to use a structured lighttechnique in which the automobile 100 illuminates an object in theenvironment with a predetermined light pattern, such as a grid orcheckerboard pattern and uses the camera 134 to detect a reflection ofthe predetermined light pattern off the object. Based on distortions inthe reflected light pattern, the automobile 100 may be configured todetermine the distance to the points on the object. The predeterminedlight pattern may comprise infrared light, or light of anotherwavelength.

The actuators 136 may, for example, be configured to modify a positionand/or orientation of the sensors.

The sensor system 104 may additionally or alternatively includecomponents other than those shown.

The control system 106 may be configured to control operation of theautomobile 100 and its components. To this end, the control system 106may include a steering unit 138, a throttle 140, a brake unit 142, asensor fusion algorithm 144, a computer vision system 146, a navigationor pathing system 148, and an obstacle avoidance system 150.

The steering unit 138 may be any combination of mechanisms configured toadjust the heading or direction of the automobile 100.

The throttle 140 may be any combination of mechanisms configured tocontrol the operating speed and acceleration of the engine/motor 118and, in turn, the speed and acceleration of the automobile 100.

The brake unit 142 may be any combination of mechanisms configured todecelerate the automobile 100. For example, the brake unit 142 may usefriction to slow the wheels/tires 124. As another example, the brakeunit 142 may be configured to be regenerative and convert the kineticenergy of the wheels/tires 124 to electric current. The brake unit 142may take other forms as well.

The sensor fusion algorithm 144 may include an algorithm (or a computerprogram product storing an algorithm) executable by the computing device111, for example. The sensor fusion algorithm 144 may be configured toaccept data from the sensor system 104 as an input. The data mayinclude, for example, data representing information sensed at thesensors of the sensor system 104. The sensor fusion algorithm 144 mayinclude, for example, a Kalman filter, a Bayesian network, or anotheralgorithm. The sensor fusion algorithm 144 further may be configured toprovide various assessments based on the data from the sensor system104, including, for example, evaluations of individual objects and/orfeatures in the environment in which the automobile 100 is located,evaluations of particular situations, and/or evaluations of possibleimpacts based on particular situations. Other assessments are possibleas well

The computer vision system 146 may be any system configured to processand analyze images captured by the camera 134 in order to identifyobjects and/or features in the environment in which the automobile 100is located, including, for example, lane information, traffic signalsand obstacles. To this end, the computer vision system 146 may use anobject recognition algorithm, a Structure from Motion (SFM) algorithm,video tracking, or other computer vision techniques. In some examples,the computer vision system 146 may additionally be configured to map theenvironment, track objects, estimate speed of objects, etc.

The navigation and pathing system 148 may be any system configured todetermine a driving path for the automobile 100. The navigation andpathing system 148 may additionally be configured to update the drivingpath dynamically while the automobile 100 is in operation. In someexamples, the navigation and pathing system 148 may be configured toincorporate data from the sensor fusion algorithm 144, the GPS module126, and one or more predetermined maps so as to determine the drivingpath for the automobile 100.

The obstacle avoidance system 150 may be any system configured toidentify, evaluate, and avoid or otherwise negotiate obstacles in theenvironment in which the automobile 100 is located.

The control system 106 may additionally or alternatively includecomponents other than those shown.

Peripherals 108 may be configured to allow the automobile 100 tointeract with external sensors, other automobiles, and/or a user. Tothis end, the peripherals 108 may include, for example, a wirelesscommunication system 152, a touchscreen 154, a microphone 156, and/or aspeaker 158.

The wireless communication system 152 may be any system configured to bewirelessly coupled to one or more other automobiles, sensors, or otherentities, either directly or via a communication network. To this end,the wireless communication system 152 may include an antenna and achipset for communicating with the other automobiles, sensors, or otherentities either directly or over an air interface. The chipset orwireless communication system 152 in general may be arranged tocommunicate according to one or more other types of wirelesscommunication (e.g., protocols) such as Bluetooth, communicationprotocols described in IEEE 802.11 (including any IEEE 802.11revisions), cellular technology (such as GSM, CDMA, UMTS, EV-DO, WiMAX,or LTE), Zigbee, dedicated short range communications (DSRC), and radiofrequency identification (RFID) communications, among otherpossibilities. The wireless communication system 152 may take otherforms as well.

The touchscreen 154 may be used by a user to input commands to theautomobile 100. To this end, the touchscreen 154 may be configured tosense at least one of a position and a movement of a user's finger viacapacitive sensing, resistance sensing, or a surface acoustic waveprocess, among other possibilities. The touchscreen 154 may be capableof sensing finger movement in a direction parallel or planar to thetouchscreen surface, in a direction normal to the touchscreen surface,or both, and may also be capable of sensing a level of pressure appliedto the touchscreen surface. The touchscreen 154 may be formed of one ormore translucent or transparent insulating layers and one or moretranslucent or transparent conducting layers. The touchscreen 154 maytake other forms as well.

The microphone 156 may be configured to receive audio (e.g., a voicecommand or other audio input) from a user of the automobile 100.Similarly, the speakers 158 may be configured to output audio to theuser of the automobile 100.

The peripherals 108 may additionally or alternatively include componentsother than those shown.

The power supply 110 may be configured to provide power to some or allof the components of the automobile 100. To this end, the power supply110 may include, for example, a rechargeable lithium-ion or lead-acidbattery. In some examples, one or more banks of batteries could beconfigured to provide electrical power. Other power supply materials andconfigurations are possible as well. In some examples, the power supply110 and energy source 120 may be implemented together, as in someall-electric cars.

The processor 113 included in the computing device 111 may comprise oneor more general-purpose processors and/or one or more special-purposeprocessors (e.g., image processor, digital signal processor, etc.). Tothe extent that the processor 113 includes more than one processor, suchprocessors could work separately or in combination. The computing device111 may be configured to control functions of the automobile 100 basedon input received through the user interface 112, for example.

The memory 114, in turn, may comprise one or more volatile and/or one ormore non-volatile storage components, such as optical, magnetic, and/ororganic storage, and the memory 114 may be integrated in whole or inpart with the processor 113. The memory 114 may contain the instructions115 (e.g., program logic) executable by the processor 113 to executevarious automobile functions, including any of the functions or methodsdescribed herein.

The components of the automobile 100 could be configured to work in aninterconnected fashion with other components within and/or outside theirrespective systems. To this end, the components and systems of theautomobile 100 may be communicatively linked together by a system bus,network, and/or other connection mechanism (not shown).

Further, while each of the components and systems is shown to beintegrated in the automobile 100, in some examples, one or morecomponents or systems may be removably mounted on or otherwise connected(mechanically or electrically) to the automobile 100 using wired orwireless connections.

The automobile 100 may include one or more elements in addition to orinstead of those shown. For example, the automobile 100 may include oneor more additional interfaces and/or power supplies. Other additionalcomponents are possible as well. In these examples, the memory 114 mayfurther include instructions executable by the processor 113 to controland/or communicate with the additional components.

FIG. 2 illustrates an example automobile 200, in accordance with anembodiment. In particular, FIG. 2 shows a Right Side View, Front View,Back View, and Top View of the automobile 200. Although automobile 200is illustrated in FIG. 2 as a car, other examples are possible. Forinstance, the automobile 200 could represent a truck, a van, asemi-trailer truck, a motorcycle, a golf cart, an off-road vehicle, or afarm vehicle, among other examples. As shown, the automobile 200includes a first sensor unit 202, a second sensor unit 204, a thirdsensor unit 206, a wireless communication system 208, and a camera 210.

Each of the first, second, and third sensor units 202-206 may includeany combination of global positioning system sensors, inertialmeasurement units, RADAR units, LIDAR units, cameras, lane detectionsensors, and acoustic sensors. Other types of sensors are possible aswell.

While the first, second, and third sensor units 202 are shown to bemounted in particular locations on the automobile 200, in some examplesthe sensor unit 202 may be mounted elsewhere on the automobile 200,either inside or outside the automobile 200. Further, while only threesensor units are shown, in some examples more or fewer sensor units maybe included in the automobile 200.

In some examples, one or more of the first, second, and third sensorunits 202-206 may include one or more movable mounts on which thesensors may be movably mounted. The movable mount may include, forexample, a rotating platform. Sensors mounted on the rotating platformcould be rotated so that the sensors may obtain information from eachdirection around the automobile 200. Alternatively or additionally, themovable mount may include a tilting platform. Sensors mounted on thetilting platform could be tilted within a particular range of anglesand/or azimuths so that the sensors may obtain information from avariety of angles. The movable mount may take other forms as well.

Further, in some examples, one or more of the first, second, and thirdsensor units 202-206 may include one or more actuators configured toadjust the position and/or orientation of sensors in the sensor unit bymoving the sensors and/or movable mounts. Example actuators includemotors, pneumatic actuators, hydraulic pistons, relays, solenoids, andpiezoelectric actuators. Other actuators are possible as well.

The wireless communication system 208 may be any system configured towirelessly couple to one or more other automobiles, sensors, or otherentities, either directly or via a communication network as describedabove with respect to the wireless communication system 152 in FIG. 1.While the wireless communication system 208 is shown to be positioned ona roof of the automobile 200, in other examples the wirelesscommunication system 208 could be located, fully or in part, elsewhere.

The camera 210 may be any camera (e.g., a still camera, a video camera,etc.) configured to capture images of the environment in which theautomobile 200 is located. To this end, the camera 210 may take any ofthe forms described above with respect to the camera 134 in FIG. 1.While the camera 210 is shown to be mounted inside a front windshield ofthe automobile 200, in other examples the camera 210 may be mountedelsewhere on the automobile 200, either inside or outside the automobile200.

The automobile 200 may include one or more other components in additionto or instead of those shown.

III. Example Methods

FIG. 3 is a flow chart of a method 300 for vision-based object detectionusing a polar grid, in accordance with an example embodiment. The method300 may include one or more operations, functions, or actions asillustrated by one or more of blocks 302-312. Although the blocks areillustrated in a sequential order, these blocks may in some instances beperformed in parallel, and/or in a different order than those describedherein. Also, the various blocks may be combined into fewer blocks,divided into additional blocks, and/or removed based upon the desiredimplementation.

In addition, for the method 300 and other processes and methodsdisclosed herein, the flowchart shows functionality and operation of onepossible implementation of present embodiments. In this regard, eachblock may represent a module, a segment, or a portion of program code,which includes one or more instructions executable by a processor forimplementing specific logical functions or steps in the process. Theprogram code may be stored on any type of computer readable medium ormemory, for example, such as a storage device including a disk or harddrive. The computer readable medium may include a non-transitorycomputer-readable medium, for example, such as computer-readable mediathat stores data for short periods of time like register memory,processor cache and Random Access Memory (RAM). The computer-readablemedium may also include non-transitory media or memory, such assecondary or persistent long term storage, like read only memory (ROM),optical or magnetic disks, compact-disc read only memory (CD-ROM), forexample. The computer-readable media may also be any other volatile ornon-volatile storage systems. The computer-readable medium may beconsidered a computer-readable storage medium, a tangible storagedevice, or other article of manufacture, for example. In addition, forthe method 300 and other processes and methods disclosed herein, eachblock in FIG. 3 may represent circuitry that is wired to perform thespecific logical functions in the process.

At block 302, the method 300 includes receiving, at a computing deviceof a first vehicle, a first image of a second vehicle captured by animage-capture device coupled to the first vehicle and a second imagecaptured by the image-capture device subsequent to capturing the firstimage, where the second vehicle has one or more flashing light signals.A controller or a computing device, such as the computing device 111 inFIG. 1, may be onboard the first vehicle or may be off-board but inwireless communication with the first vehicle, for example. Also, thecomputing device may be configured to control the first vehicle in anautonomous or semi-autonomous operation mode.

A camera, such as the camera 134 in FIG. 1 or the camera 210 in FIG. 2or any other image-capture device (e.g., a LIDAR unit such as the LIDARunit 132 depicted in FIG. 1), may be coupled to the first vehicle andmay be in communication with the computing device. The image-capturedevice may be configured to capture images or a video of a road oftravel of the first vehicle and vicinity of the road. The computingdevice may be configured to receive a sequence of images or a video andidentify, using image processing techniques for example, objectsdepicted in the image or the video. Examples of objects may includevehicles, moving or static objects, traffic signs, obstacles on theroad, pedestrians, lane markers, etc.

FIG. 4 illustrates an image 400 depicting an emergency vehicle 402, inaccordance with an example embodiment. The image 400 may be capturedfrom a camera coupled to the first vehicle (not shown), for example. Theimage 400 depicts a vicinity of the first vehicle and includes theemergency vehicle 402, which is a fire truck in this case. The emergencyvehicle 402 is characterized by four flashing lights 401A, 401B, 401C,and 401D as depicted in FIG. 4. Although a fire truck is used herein toillustrate the method 300, the method 300 can be implemented to identifyany other type of vehicle having a flashing light.

The computing device may receive a second image (not shown) capturedsubsequent to capturing the first image. The computing device mayreceive a sequence of images captured depicting the vehicle 402 and maybe configured to monitor the vehicle 402 across the sequence of images(or frames). In one example, the images may be captured at a particularfrequency (e.g., 10 Hz or 10 pictures per second or any other frequencysuitable for a given type of flashing lights). In an example, the firstimage and the second image may be captured consecutively. In anotherexample, the first image and the second image may not be capturedconsecutively. For instance, if a given type of vehicle has a flashinglight that is flashing at a slow frequency, the computing device may beconfigured to detect the vehicle 402 in a first image, skip over anumber of images in the sequence of images, and detect the vehicle 402in a second image captured subsequently but not consecutive to the firstimage. The method 300 illustrates detecting a number and type offlashing lights (such as the flashing lights 401A, 401B, 401C, and 401D)associated with the emergency vehicle 402, determining the type of theemergency vehicle 402 based on the number and type of flashing lights,and controlling the first vehicle based on the determination.

Referring back to FIG. 3, at block 304, the method 300 includesdetermining, in the first image and the second image, an image regionthat bounds the second vehicle such that the image region substantiallyencompasses the second vehicle. FIG. 5 illustrates an image regionbounding the emergency vehicle 402, in accordance with an exampleembodiment. For example, the computing device may be configured todetermine an image region that encompasses pixels in the first image andthe second image that represent the emergency vehicle 402. For example,the image region may be determined based on detecting edges orboundaries of the emergency vehicle 402 and determining an area in theimage that surrounds or encompasses the emergency vehicle 402accordingly. The image region may have any type of a geometric shape.For example, referring to FIG. 5, the image region may take a shape of abox 500 that substantially encompasses the emergency vehicle 402.However, the shape can be any type of geometric shape (e.g., a circle,an ellipse, a square, etc.).

The word “substantially” is used herein to indicate, for instance, thatthe image region, such as the box 500, encompasses a certain percentageof image space or percentage of pixels that represent the emergencyvehicle 402. For instance, the image region may substantially encompassthe emergency vehicle 402 or the pixels if the image region includes atleast 95% of the pixels representing the emergency vehicle 402 in theimage 400. In another example, the image region may be larger than theemergency vehicle 402 such that the emergency vehicle 402 is whollysurrounded by the image region, i.e., the box 500. These example shapesand example percentages are used as examples for illustration only, andother shapes and percentages can be used as well.

Tracing the emergency vehicle 402 across multiple images may bedifficult. In one example, as the emergency vehicle 402 moves relativeto the first vehicle, the emergency vehicle 402 changes position, i.e.,shifts horizontally, in sequential images or frames. In another example,the box 500 may not be determined accurately in each image (e.g., thebox 500 may be slightly shifted relative to location of the emergencyvehicle 402 in a given image). As still another example, assuming thatthe first vehicle approaches the emergency vehicle 402, a size of thearea of a given image depicting the emergency vehicle 402 may increasein size. Similarly, a size of the area of a given image depicting theemergency vehicle 402 may decrease if the emergency vehicle 402 movesfarther away from the first vehicle. Such changes in size and positionof the emergency vehicle 402 may lead to inaccurate determination of thebox 500. Slight errors in position or size may accumulate over asequence of images and may affect accuracy of tracing the emergencyvehicle 402 across the sequence of images, and may thus affectestimating the type of emergency vehicle 402 and the number and types offlashing lights of the emergency vehicle 402. As described below, atblock 306, the method 300 illustrates partitioning the box 500 using apolar grid into a plurality of polar bins to enhance accuracy ofestimating the type of the emergency vehicle 402 and the number andtypes of flashing lights.

Referring back to FIG. 3, at block 306, the method 300 includesdetermining a polar grid that partitions the image region in the firstimage and the second image into a plurality of polar bins, where eachpolar bin includes a sector of the image region. To enhance objectdetection and tracing, the computing device may transform the imageregion encompassing the second vehicle (such as the box 500 encompassingthe emergency vehicle 402) into a polar grid or an angular region.

FIG. 6 illustrates a polar grid configured to partition the image regioninto a plurality of polar bins, in accordance with an exampleembodiment. FIG. 6 depicts a polar grid 600 that partitions the box 500into a plurality of polar bins, such as polar bin 602. Each polar binincludes or encompasses a sector of the box 500. As an example, eachpolar bin may be defined by two lines, such as lines 604 and 606,extending from a center region of the box 500 to about a boundary of thebox 500. The center region may be defined by a group of pixels within athreshold number of pixels (or a threshold distance) from a center ofthe box 500, for example. Similarly, the lines extend to about aboundary of the box indicates that the lines extend to within athreshold number of pixels or threshold distance from a boundary of thebox 500. However, the polar bins may take other shapes as well. Thethreshold number of pixels of threshold distance may be predefine orpreviously determined.

FIG. 7 illustrates example types of polar bins, in accordance with anexample embodiment. For example, FIG. 7 depicts a triangular polar bin700 defined by two lines 701A and 701B extending from a center region702 bounded by circle to about a boundary of the box 500. As describedabove, the center region 702 may be defined to be a region that iswithin a particular number of pixels or a particular threshold distancefrom a center of the box 500. Similarly, lines 701A and 701B extend toabout a boundary of the box 500 indicating that the lines 701A and 701Bextend to within a threshold number of pixels or threshold distance froma boundary of the box 500. Although the center region 702 is shown as acircle, the center region may take other forms and geometric shapes aswell (e.g., a square, rectangle, ellipse, etc.).

FIG. 7 also depicts polar bin 704 shaped as a quadrilateral. The polarbin 704 is defined by two lines 705A and 705B extending from the centerregion 702 to about respective boundaries of the box 500. Because eachline extends to a different boundary, the polar bin 704 takes the shapeof a quadrilateral instead of a triangle. However, polar bin 706, forexample, is defined as a triangle despite being defined by two lines707A and 707B extending from the center region 702 to differentboundaries of the box 500. In this case a small region 708 is notincluded in the polar bin 706. FIG. 7 also depicts polar bin 710 definedby two lines 711A and 711B extending from the center region 702 but donot reach the boundary of the box 500 and bounded by a curved line 711Cinstead of a straight line as polar bins 700, 704, and 706. These polarbin shapes are examples for illustration and other shapes are alsopossible.

Partitioning the box 500 into a plurality of polar bins may improveaccuracy of tracing a given object. A polar bin, such as the polar bin700 is characterized by encompassing a small area of an image close to acenter of the box 500, yet encompassing a larger area of the image closeto the boundary of the box 500 (i.e., the polar bin 700 encompasses aprogressively larger area of the image or the box 500 in a direction ofarrow 712). In examples, flashing lights are disposed at extremities oredges of a given vehicle such as the emergency vehicle 402. Thus, theflashing lights are likely to be included within a portion of the polarbin that encompasses a larger image area. Tracing a flashing light inthe larger area of the polar bin may be tolerant to position errors inlocation of a flashing light from one image to a subsequent image (orfrom a polar bin in a given image to a corresponding polar bin in asubsequent image). Even if a location of a flashing light moves from agiven location in a polar bin in a given image to a different locationwithin a corresponding polar bin in a subsequent image, the flashinglight may still be traced accurately because the flashing light mayremain within the corresponding polar bin as opposed to shifting to adifferent polar bin.

Furthermore, the polar bins may be non-uniform in size. Some polar binsmay be larger than others. For example, the polar bin 700 is larger thanthe polar bin 706 as shown in FIG. 7. In an example, if the computingdevice determines that several polar bins do not include image regionswith potential flashing lights, the computing device may merge suchpolar bins into a single polar bin to enhance computational efficiency.Enhancing computational efficiency enables real-time analysis anddetermination of types of vehicles and flashing lights.

Further, as described above, a given polar bin includes a smaller imageregion in a part of the polar bin that is closer to a center of the box500 that is unlikely to include a flashing light. Thus, the computingdevice may determine an area around a center of the box 500 that definesregions within the polar bins of the polar grid that may be disregardedwhen identifying flashing lights. Thus, the computing device may nottake into consideration portions of the polar bins that lay within thatarea when analyzing a given image to identify flashing lights. In thismanner, the computing device may enhance computational efficiencyfurther by reducing amount of data to be analyzed. The area around thecenter of the box 500 may take any geometric shape (e.g., a circle,square, rectangle, etc.). In one example, the area may be related to thecenter region 702. In another example, the area may not be related tothe center region 702, i.e., the area may have a different shape and/orsize compared to the center region 702.

Referring back to FIG. 6, the computing device may be configured todetermine the polar grid 600 configured to partition the box 500 into aplurality of polar bins having shapes similar to shapes of any of thepolar bins 700, 704, 706, and 710 or a combination thereof. The polargrid can partition the box 500 to any number of polar bins. As anexample for illustration, FIG. 6 depicts forty four polar bins. However,any other number of polar bins can be used (e.g., twelve bins).

In an example, the computing device may assign each pixel in the imageregion within the box 500 to one of the polar bins. FIG. 8 illustratesassigning a pixel within the image region to a polar bin, in accordancewith an example embodiment. FIG. 8 depicts the box 500 and Cartesiancoordinate system 800. As an example for illustration, the computingdevice may determine a polar grid that partitions the box 500 into fourpolar bins 802, 804, 806, and 808. Each polar bin may be assigned anindex number. For instance, polar bin 802 may be assigned an indexnumber “0,” polar bin 804 may be assigned an index number “1,” polar bin806 may be assigned an index number “2,” and polar bin 808 may beassigned an index number “3.” In this example, each pixel within the box500 such as pixel 810 having coordinates (x,y) can be assigned to one ofthe four polar bins 802, 804, 806, and 808 using the following equation:

$\begin{matrix}{{index} = {{floor}\left( {N^{*}\frac{{{atan}\mspace{14mu} 2\left( \frac{y}{x} \right)} + \pi}{2^{*}\pi}} \right)}} & {{Equation}\mspace{14mu}(1)}\end{matrix}$where N is the number of polar bins (4 in this example), “floor” is afunction that is configured to map a real number to the largest previousinteger, respectively, and where “atan 2” is four-quadrant inversetangent (arctangent) function of coordinates y and x. The “floor”function can also be referred to as the greatest integer function. As anexample, consider the fraction 12/5, which equals 2.4. The “floor” of(12/5) equals 2. Equation (1) returns an index number 0, 1, 2, or 3 thatdetermines whether the pixel 810 belongs to polar bin 802, 804, 806, or808, respectively.

It is noted that equation (1) is an example for illustration only and isnot limiting. Other functions can be implemented to assign the pixel 810to a polar bin. Also, the four polar bins are used herein as an examplefor illustration. For instance the polar grid may partition the box 500into a larger number of bins (e.g., twelve polar bins). The polar binsmay have different shapes and configurations. In these examples,different equations or algorithms may be used to assign the pixel 810 toa particular polar bin.

In this manner, the image region or the box 500 encompassing theemergency vehicle 402 in the first image, and the corresponding regionin the second image may be partitioned into a respective plurality ofbins. Thus, a polar bin in the polar grid of the first image has acorresponding polar bin in the polar grid of the second image, and insubsequent images captured for the emergency vehicle 402.

Referring back to FIG. 3, at block 308, the method 300 includesidentifying, based on a comparison of image content of polar bins in thefirst image to image content of corresponding polar bins in the secondimage, one or more portions of image data exhibiting a change in colorand a change in brightness between the first image and the second image.The computing device may identify bright lights within each polar gridbin in a first image. For example, the computing device may determine anumber of image portions that exhibit characteristics of a bright lighthaving an intensity greater than a threshold intensity. As an example,the threshold intensity may represent a predetermined or predefinedbrightness threshold indicative of a minimum brightness associated withflashing lights an emergency vehicle, a vehicular turn signal, brakelights, etc. Within each corresponding polar bin in the second image,the computing device may identify image portions corresponding to theimage portions identified in the first image. The computing device maythen determine a change in color and a change in brightness for theimage portions from the polar bin in the first image to thecorresponding polar bin in the second image.

In an example, the computing device may be configured to perform apixel-level comparison between pixels in a polar in in the first imageand pixels in a corresponding polar bin in the second images to identifydifferences or changes in color and intensities. In another example, todetermine a difference between content of a polar bin in the first imageand content of a corresponding polar bin in the second image, thecomputing device may be configured to subtract the content of polar binfrom the content of the corresponding polar bin, or vice versa.

In examples, each pixel within a polar bin may be assigned numericalvalues that represent color and intensity attributes of the pixel.Intensity may refer to a brightness level of a given pixel, for example.Brightness may be an attribute of visual perception in which a sourceappears to be radiating or reflecting light. In other words, brightnessmay be a perception produced by luminance of a visual target, such aslight of a vehicular signal. Example cylindrical-coordinaterepresentations of pixels in a Red, Green, Blue (RGB) model may includeHSL representation (hue, saturation, and lightness), or HSV (hue,saturation, and value) representation. The HSV representation can alsobe referred to as HSB, where B stands for Brightness.

Subtracting content of a polar bin in the first image from content of acorresponding polar bin in the second image may include subtracting HSVor HSL values of pixels of the first image from respective HSV or HSLvalues of respective pixels of the second image, for example. HSL andHSV are used herein as examples only, and any other numerical orqualitative representation can be used to assign intensity and colorattributes or characteristics to pixels within a polar bin of a givenimage.

In an example, the computing device may be configured to generate apolar histogram for the image region, e.g., the box 500. For each polarbin, the computing device may assign an index number to an identifiedbright region, and store an intensity value and/or a color value for theidentified bright region. The computing device may generate a polarhistogram for the image region indicative of the number of identifiedbright regions, and associated intensity and color values. The computingdevice may be configured to generate a respective polar histogram forimages of the sequence of images (e.g., for the first image and thesecond image). Based on comparing the polar histograms of a given imageand a subsequent image, the computing device may be configured todetermine changes in color and intensity associated with content of aparticular polar bin in the given image and a corresponding polar bin inthe subsequent image.

Referring back to FIG. 3, at block 310, the method 300 includesdetermining a type of the one or more flashing light signals of thesecond vehicle and a type of the second vehicle based on (i) a number ofportions of image data exhibiting the change in color and the change inbrightness, (ii) the color of the one or more portions, (iii) and thebrightness of the one or more portions.

FIG. 9 illustrates determining a type of the emergency vehicle 402, inaccordance with an example embodiment. Based on comparing content ofpolar bins in the first image with content of corresponding polar binsin the second image, the computing device may be configured to determinea number of portions of image data exhibiting a change in color and/orbrightness. Thus, the computing device may estimate a number of flashinglights associated with the emergency vehicle 402.

In examples, the computing device may trace the flashing lightsidentified in the polar bins across a sequence of images and determinecharacteristics or parameters of repetitive or cyclical change in coloror brightness of the flashing lights across the images and accordinglydetermine types of the flashing lights. For instance, the computingdevice may determine one or more temporal characteristics indicative ofa frequency of the change in brightness of image portions representingflashing lights in the images. The computing device may, for example,trace oscillation in intensity values overtime for a given flashinglight. The oscillation may match behavior of a particular type offlashing light (e.g., match a blinking rate of vehicular turn signalthat turns ‘on’ and ‘off’ at a certain frequency, match blinkingfrequency of flashing light of a fire truck, or match a blinkingfrequency of a police car blue flashing light, etc.).

In this manner, the computing device may determine a number and type offlashing lights associated with the emergency vehicle 402. Based ondetermining the number and type of flashing lights, the computing devicemay determine the type of the emergency vehicle. As shown in FIG. 9, theemergency vehicle 402 has the four flashing lights 401A, 401B, 401C, and401D. The flashing lights 401A, 401B, 401C, and 401D may have red/orangecolors, for example. Based on determining the number of the flashinglights 401A, 401B, 401C, and 401D and the associated colors, thecomputing device may determine that the emergency vehicle 402 is a firetruck.

FIG. 9 depicts polar bins having different sizes, i.e., non-uniform insize. Some polar bins may be larger than others. For example, polar bin900 is larger than the polar bin 602. As described with respect to FIG.7, if the computing device determines that several polar bins do notinclude image regions representing potential flashing lights, thecomputing device may merge such polar bins into a single polar bin toenhance computational efficiency. Thus, as an example, the polar bin 900results from merging several polar bins similar to the polar bin 602.

Also, as described above with respect to FIG. 7, a given polar binencompasses a smaller image area in a part of the polar bin that iscloser to the center of the box 500 that is unlikely to include aflashing light. Thus, the computing device may determine an area, suchas area 902 defined by a circle, that defines regions within the polarbins of the polar grid that may be disregarded when identifying flashinglights. In other words, the computing device may not take intoconsideration portions of the polar bins that lay within the area 902when analyzing a given image to identify flashing lights. In thismanner, the computing device may enhance computational efficiencyfurther by reducing amount of data to be analyzed. However, in otherexamples, the computing device may take the entire content of the polarbins into consideration when identifying flashing lights.

Although the method 300 is described in the context of detecting anemergency vehicle, the method 300 can be used to detect any other typeof light signals such as a brake light, headlamp light, an auxiliarylamp light, or turn signals. FIG. 10 illustrates other examples ofdetermining types of vehicles, in accordance with an example embodiment.FIG. 10 depicts a vehicle 1000 having two flashing lights 1002A and1002B. The computing device may identify the flashing lights 1002A and1002B and determine that they exhibit a blue color. Accordingly, thecomputing device may determine that the vehicle 1000 is a policevehicle, for example. Similarly, the computing device may detect avehicle 1004, and identify a flashing vehicular turn signal 1006 basedon determining a color of the flashing vehicular turn signal 1006 and ablinking rate of the vehicular turn signal 1006, for example.

Referring back to FIG. 3, at block 312, the method 300 includesproviding, by the computing device, instructions to control the firstvehicle based on the type of the second vehicle and the type of the oneor more flashing light signals. A control strategy and driving behaviorof the first vehicle may be influenced by types of vehicles in avicinity of the first vehicle. For example, a more defensive (e.g.,cautious) behavior may be selected and implemented by the computingdevice to control the first vehicle if an emergency vehicle (e.g., afire truck, ambulance, police vehicle) is in a vicinity of the firstvehicle. As another example, the computing device may be configured todetect presence of another vehicle with a flashing turning signal andmay be configured to control the first vehicle based on the detection.

The control system of the first vehicle may support multiple controlstrategies and associated driving behaviors that may be predetermined oradaptive to changes in a driving environment of the first vehicle.Generally, a control strategy may comprise sets of rules associated withtraffic interaction in various driving contexts. The control strategymay comprise rules that determine a speed of the first vehicle and alane that the first vehicle may travel on while taking into accountsafety and traffic rules and concerns (e.g., presence of emergencyvehicles, vehicles having active turn signals, etc.). For instance, thecomputing device may be configured to determine that a given vehicle infront of the first vehicle has an active turn signal to the left orright. The given vehicle may, for example, change lines to a lanecurrently occupied by the first vehicle, and the computing device mayprovide instructions for the first vehicle to slow down or change lanes.As another example, the computing device may determine that there is afire truck in the vicinity of the first vehicle and cause the firstvehicle to stop and allow the fire truck to pass. Thus, the computingdevice may be configured to select a control strategy comprising rulesfor actions that control the first vehicle based on detecting types ofvehicles in the vicinity and types of flashing lights associated withthe vehicles.

In an example, a given control strategy may comprise a program orcomputer instructions that characterize actuators controlling the firstvehicle (e.g., throttle, steering gear, brake, accelerator, ortransmission shifter) based on determining that there is an emergencyvehicle in the vicinity of the first vehicle or that a given vehicle hasan active turn signal. The given control strategy may include actionsets ranked by priority, and the action sets may include alternativeactions that the first vehicle may be configured to take to accomplish atask (e.g., driving from one location to another). The alternativeactions may be ranked based on whether an emergency vehicle is presentin a vicinity of the first vehicle, for example.

As another example, providing instructions to control the first vehiclemay comprise determining a desired path of the first vehicle based ondetermining whether a given vehicle in a vicinity of the first vehicleis an emergency vehicle or has an active turn signal. In this example,the computing device may be configured to change the path of the firstvehicle to take into account the likelihood that the given vehicle maychange lanes and influence a current path of the first vehicle.

These control actions and driving situations are for illustration only.Other actions and situations are possible as well. In one example, thecomputing device may be configured to control the vehicle based on themodified control strategy as an interim control until a human driver cantake control of the vehicle.

In some embodiments, the disclosed methods may be implemented ascomputer program instructions encoded on a computer-readable storagemedia in a machine-readable format, or on other non-transitory media orarticles of manufacture. FIG. 11 is a schematic illustrating aconceptual partial view of an example computer program product 1100 thatincludes a computer program for executing a computer process on acomputing device, arranged according to at least some embodimentspresented herein. In one embodiment, the example computer programproduct 1100 is provided using a signal bearing medium 1101. The signalbearing medium 1101 may include one or more program instructions 1102that, when executed by one or more processors (e.g., processor 113 inthe computing device 111) may provide functionality or portions of thefunctionality described above with respect to FIGS. 1-10. Thus, forexample, referring to the embodiments shown in FIG. 3, one or morefeatures of blocks 302-312 may be undertaken by one or more instructionsassociated with the signal bearing medium 1101. In addition, the programinstructions 1102 in FIG. 11 describe example instructions as well.

In some examples, the signal bearing medium 1101 may encompass acomputer-readable medium 1103, such as, but not limited to, a hard diskdrive, a Compact Disc (CD), a Digital Video Disk (DVD), a digital tape,memory, etc. In some implementations, the signal bearing medium 1101 mayencompass a computer recordable medium 1104, such as, but not limitedto, memory, read/write (R/W) CDs, R/W DVDs, etc. In someimplementations, the signal bearing medium 1101 may encompass acommunications medium 1105, such as, but not limited to, a digitaland/or an analog communication medium (e.g., a fiber optic cable, awaveguide, a wired communications link, a wireless communication link,etc.). Thus, for example, the signal bearing medium 1101 may be conveyedby a wireless form of the communications medium 1105 (e.g., a wirelesscommunications medium conforming to the IEEE 802.11 standard or othertransmission protocol).

The one or more programming instructions 1102 may be, for example,computer executable and/or logic implemented instructions. In someexamples, a computing device such as any of the computing devicesdescribed with respect to FIGS. 1-10 may be configured to providevarious operations, functions, or actions in response to the programminginstructions 1102 conveyed to the computing device by one or more of thecomputer readable medium 1103, the computer recordable medium 1104,and/or the communications medium 1105. It should be understood thatarrangements described herein are for purposes of example only. As such,those skilled in the art will appreciate that other arrangements andother elements (e.g. machines, interfaces, functions, orders, andgroupings of functions, etc.) can be used instead, and some elements maybe omitted altogether according to the desired results. Further, many ofthe elements that are described are functional entities that may beimplemented as discrete or distributed components or in conjunction withother components, in any suitable combination and location.

IV. Conclusion

It should be understood that arrangements described herein are forpurposes of example only. As such, those skilled in the art willappreciate that other arrangements and other elements (e.g., machines,interfaces, functions, orders, and groupings of functions, etc.) can beused instead, and some elements may be omitted altogether according tothe desired results. Further, many of the elements that are describedare functional entities that may be implemented as discrete ordistributed components or in conjunction with other components, in anysuitable combination and location.

While various aspects and embodiments have been disclosed herein, otheraspects and embodiments will be apparent to those skilled in the art.The various aspects and embodiments disclosed herein are for purposes ofillustration and are not intended to be limiting, with the true scopeand spirit being indicated by the following claims, along with the fullscope of equivalents to which such claims are entitled. It is also to beunderstood that the terminology used herein is for the purpose ofdescribing particular embodiments only, and is not intended to belimiting.

Where example embodiments involve information related to a person or adevice of a person, some examples may include privacy controls. Suchprivacy controls may include, at least, anonymization of deviceidentifiers, transparency and user controls, including functionalitythat would enable users to modify or delete information relating to theuser's use of a product.

Further, in situations in where embodiments discussed herein collectpersonal information about users, or may make use of personalinformation, the users may be provided with an opportunity to controlwhether programs or features collect user information (e.g., informationabout a user's medical history, social network, social actions oractivities, profession, a user's preferences, or a user's currentlocation), or to control whether and/or how to receive content from thecontent server that may be more relevant to the user. In addition,certain data may be treated in one or more ways before it is stored orused, so that personally identifiable information is removed. Forexample, a user's identity may be treated so that no personallyidentifiable information can be determined for the user, or a user'sgeographic location may be generalized where location information isobtained (such as to a city, ZIP code, or state level), so that aparticular location of a user cannot be determined. Thus, the user mayhave control over how information is collected about the user and usedby a content server.

What is claimed is:
 1. A method comprising: receiving, at a computingdevice of a first vehicle, a first image of a second vehicle captured byan image-capture device coupled to the first vehicle and a second imagecaptured by the image-capture device subsequent to capturing the firstimage, wherein the second vehicle has one or more flashing lightsignals; determining a polar grid that partitions an image region in thefirst image and the second image into a plurality of polar bins, whereineach polar bin of the plurality of polar bins is defined by two linesextending from a center portion of the image region to about a boundaryof the image region; identifying, based on a comparison of image contentof polar bins in the first image to image content of corresponding polarbins in the second image, one or more portions of image data exhibitinga change in brightness between the first image and the second image;determining a type of the one or more flashing light signals of thesecond vehicle based on the one or more portions exhibiting the changein brightness; and controlling the first vehicle based on the type ofthe one or more flashing light signals.
 2. The method of claim 1,wherein identifying the one or more portions of image data comprises:within each polar bin in the first image, identifying image portionsthat exhibit characteristics of a bright light having an intensitygreater than a threshold intensity; within each corresponding polar binin the second image, identifying corresponding image portions; anddetermining the change in brightness for the image portions from thepolar bin in the first image to the corresponding polar bin in thesecond image.
 3. The method of claim 1, wherein the polar bins arenon-uniform in size.
 4. The method of claim 1, determining that severalpolar bins of the plurality of polar bins are void of image pixelsexhibiting the change in brightness between the first image and thesecond image; and merging the several polar bins into a single polarbin.
 5. The method of claim 1, further comprising: wherein the imageregion bounds the second vehicle such that the image regionsubstantially encompasses the second vehicle.
 6. The method of claim 5,wherein the second vehicle is moving relative to the first vehicle suchthat the image region that bounds the second vehicle in the first imageis different from a respective image region that bounds the secondvehicle in the second image.
 7. The method of claim 1, furthercomprising: assigning each pixel in the image region to a respectivepolar bin of the polar bins.
 8. The method of claim 1, furthercomprising: determining a type of the second vehicle based on (i) anumber of portions of image data exhibiting the change in brightness,(ii) color of the one or more portions, and (iii) the brightness of theone or more portions, wherein controlling the first vehicle is furtherbased on the type of the second vehicle.
 9. The method of claim 1,wherein determining the type of a given flashing light signal of the oneor more flashing light signals comprises determining that the givenflashing light is a vehicular turn signal or a brake light.
 10. Themethod of claim 1, wherein the first image and the second image arecaptured consecutively.
 11. The method of claim 1, wherein the computingdevice is configured to control the first vehicle in an autonomousoperation mode.
 12. A non-transitory computer readable medium havingstored thereon executable instructions that, upon execution by acomputing device of a first vehicle, cause the computing device toperform operations comprising: receiving a first image of a secondvehicle captured by an image-capture device coupled to the first vehicleand a second image captured by the image-capture device subsequent tocapturing the first image, wherein the second vehicle has one or moreflashing light signals; determining a polar grid that partitions animage region in the first image and the second image into a plurality ofpolar bins, wherein each polar bin of the plurality of polar bins isdefined by two lines extending from a center portion of the image regionto about a boundary of the image region; identifying, based on acomparison of image content of polar bins in the first image to imagecontent of corresponding polar bins in the second image, one or moreportions of image data exhibiting a change in brightness between thefirst image and the second image; determining a type of the one or moreflashing light signals of the second vehicle based on the one or moreportions exhibiting the change in brightness; and controlling the firstvehicle based on the type of the one or more flashing light signals. 13.The non-transitory computer readable medium of claim 12, whereinidentifying the one or more portions of image data comprises: withineach polar bin in the first image, identifying image portions thatexhibit characteristics of a bright light having an intensity greaterthan a threshold intensity; within each corresponding polar bin in thesecond image, identifying corresponding image portions; and determiningthe change in brightness for the image portions from the polar bin inthe first image to the corresponding polar bin in the second image. 14.The non-transitory computer readable medium of claim 12, wherein thepolar bins are non-uniform in size.
 15. The non-transitory computerreadable medium of claim 12, wherein the operations further comprise:determining that several polar bins of the plurality of polar bins arevoid of image pixels exhibiting the change in brightness between thefirst image and the second image; and merging the several polar binsinto a single polar bin.
 16. The non-transitory computer readable mediumof claim 12, wherein the operations further comprise: assigning eachpixel in the image region to a respective polar bin of the polar bins.17. A system comprising: an image-capture device coupled to a firstvehicle; at least one processor in communication with the image-capturedevice; and a memory having stored thereon executable instructions that,upon execution by the at least one processor, cause the at least oneprocessor to perform operations comprising: receiving a first image of asecond vehicle captured by the image-capture device and a second imagecaptured by the image-capture device subsequent to capturing the firstimage, wherein the second vehicle has one or more flashing lightsignals, determining a polar grid that partitions an image region in thefirst image and the second image into a plurality of polar bins, whereineach polar bin of the plurality of polar bins is defined by two linesextending from a center portion of the image region to about a boundaryof the image region, identifying, based on a comparison of image contentof polar bins in the first image to image content of corresponding polarbins in the second image, one or more portions of image data exhibitinga change in brightness between the first image and the second image,determining a type of the one or more flashing light signals of thesecond vehicle based on the one or more portions exhibiting the changein brightness, and controlling the first vehicle based on the type ofthe one or more flashing light signals.
 18. The system of claim 17,wherein identifying the one or more portions of image data comprises:within each polar bin in the first image, identifying image portionsthat exhibit characteristics of a bright light having an intensitygreater than a threshold intensity; within each corresponding polar binin the second image, identifying corresponding image portions; anddetermining the change in brightness for the image portions from thepolar bin in the first image to the corresponding polar bin in thesecond image.
 19. The system of claim 17, wherein the operations furthercomprise: determining that several polar bins of the plurality of polarbins are void of image pixels exhibiting the change in brightnessbetween the first image and the second image; and merging the severalpolar bins into a single polar bin such that the polar bins becomenon-uniform in size.
 20. The system of claim 17, wherein the operationsfurther comprise: assigning each pixel in the image region to arespective polar bin of the polar bins.